Camera array for a mediated-reality system

ABSTRACT

A camera array for a mediated-reality system includes a plurality of hexagonal cells arranged in a honeycomb pattern in which a pair of inner cells include respective edges adjacent to each other and a pair of outer cells are separated from each other by the inner cells. A plurality of cameras are mounted within each of the plurality of hexagonal cells. The plurality of cameras include at least one camera of a first type and at least one camera of a second type. The camera of the first type may have a longer focal length than the camera of the second type.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/737,791 filed on Sep. 27, 2018, which is incorporated by referenceherein.

BACKGROUND Technical Field

The disclosed embodiments relate generally to a camera array, and morespecifically, to a camera array for generating a virtual perspective ofa scene for a mediated-reality viewer.

Description of the Related Art

In a mediated reality system, an image processing system adds,subtracts, or modifies visual information representing an environment.For surgical applications, a mediated reality system may enable asurgeon to view a surgical site from a desired perspective together withcontextual information that assists the surgeon in more efficiently andprecisely performing surgical tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example embodiment of animaging system.

FIG. 2 is an example of a surgical environment employing the imagingsystem for mediated-reality assisted surgery.

FIG. 3 is simplified cross-sectional view of an example embodiment of acamera array.

FIG. 4 is a detailed bottom view of an example embodiment of a cameraarray.

FIG. 5 is a top perspective view of an example embodiment of a cameraarray.

DETAILED DESCRIPTION

The figures and the following description relate to preferredembodiments by way of illustration only. It should be noted that fromthe following discussion, alternative embodiments of the structures andmethods disclosed herein will be readily recognized as viablealternatives that may be employed without departing from the principlesof what is claimed.

Reference will now be made in detail to several embodiments, examples ofwhich are illustrated in the accompanying figures. It is noted thatwherever practicable similar or like reference numbers may be used inthe figures and may indicate similar or like functionality. The figuresdepict embodiments of the disclosed system (or method) for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles described herein.

Overview

A camera array includes a plurality of hexagonal cells arranged in ahoneycomb pattern in which a pair of inner cells include respectiveedges adjacent to each other and a pair of outer cells are separatedfrom each other by the inner cells. A plurality of cameras is mountedwithin each of the plurality of hexagonal cells. The plurality ofcameras includes at least one camera of a first type and at least onecamera of a second type. For example, the camera of the first type mayhave a longer focal length than the camera of the second type. Theplurality of cameras within each of the plurality of hexagonal cells arearranged in a triangular grid approximately equidistant from neighboringcameras. In an embodiment, at least one camera of the second type withineach of the plurality of hexagonal cells is at a position further fromor equidistant from a center point of the camera array relative tocameras of the first type.

Mediated-Reality System

FIG. 1 illustrates an example embodiment of a mediated-reality system100. The mediated-reality system 100 comprises an image processingdevice 110, a camera array 120, a display device 140, and an inputcontroller 150. In alternative embodiments, the mediated-reality system100 may comprise additional or different components.

The camera array 120 comprises a plurality of cameras 122 (e.g., acamera 122-1, a camera 122-2, . . . , a camera 122-N) that each capturerespective images of a scene 130. The cameras 122 may be physicallyarranged in a particular configuration as described in further detailbelow such that their physical locations and orientations relative toeach other are fixed. For example, the cameras 122 may be structurallysecured by a mounting structure to mount the cameras 122 at predefinedfixed locations and orientations. The cameras 122 of the camera array120 may be positioned such that neighboring cameras may shareoverlapping views of the scene 130. The cameras 122 in the camera array120 may furthermore be synchronized to capture images of the scene 130substantially simultaneously (e.g., within a threshold temporal error).The camera array 120 may furthermore comprise one or more projectors 124that projects a structured light pattern onto the scene 130. The cameraarray 120 may furthermore comprise one or more depth sensors 126 thatperform depth estimation of a surface of the scene 150.

The image processing device 110 receives images captured by the cameraarray 120 and processes the images to synthesize an output imagecorresponding to a virtual camera perspective. Here, the output imagecorresponds to an approximation of an image of the scene 130 that wouldbe captured by a camera placed at an arbitrary position and orientationcorresponding to the virtual camera perspective. The image processingdevice 110 synthesizes the output image from a subset (e.g., two ormore) of the cameras 122 in the camera array 120, but does notnecessarily utilize images from all of the cameras 122. For example, fora given virtual camera perspective, the image processing device 110 mayselect a stereoscopic pair of images from two cameras 122 that arepositioned and oriented to most closely match the virtual cameraperspective.

The image processing device 110 may furthermore perform a depthestimation for each surface point of the scene 150. In an embodiment,the image processing device 110 detects the structured light projectedonto the scene 130 by the projector 124 to estimate depth information ofthe scene. Alternatively, or in addition, the image processing device110 includes dedicated depth sensors 126 that provide depth informationto the image processing device 110. In yet other embodiments, the imageprocessing device 110 may estimate depth only from multi-view image datawithout necessarily utilizing any projector 124 or depth sensors 126.The depth information may be combined with the images from the cameras122 to synthesize the output image as a three-dimensional rendering ofthe scene as viewed from the virtual camera perspective.

In an embodiment, functions attributed to the image processing device110 may be practically implemented by two or more physical devices. Forexample, in an embodiment, a synchronization controller controls imagesdisplayed by the projector 124 and sends synchronization signals to thecameras 122 to ensure synchronization between the cameras 122 and theprojector 124 to enable fast, multi-frame, multi-camera structured lightscans. Additionally, this synchronization controller may operate as aparameter server that stores hardware specific configurations such asparameters of the structured light scan, camera settings, and cameracalibration data specific to the camera configuration of the cameraarray 120. The synchronization controller may be implemented in aseparate physical device from a display controller that controls thedisplay device 140, or the devices may be integrated together.

The virtual camera perspective may be controlled by an input controller150 that provides a control input corresponding to the location andorientation of the virtual imager perspective. The output imagecorresponding to the virtual camera perspective is outputted to thedisplay device 140 and displayed by the display device 140. The imageprocessing device 110 may beneficially process received inputs from theinput controller 150 and process the captured images from the cameraarray 120 to generate output images corresponding to the virtualperspective in substantially real-time as perceived by a viewer of thedisplay device 140 (e.g., at least as fast as the frame rate of thecamera array 120).

The image processing device 110 may comprise a processor and anon-transitory computer-readable storage medium that stores instructionsthat when executed by the processor, carry out the functions attributedto the image processing device 110 as described herein.

The display device 140 may comprise, for example, a head-mounted displaydevice or other display device for displaying the output images receivedfrom the image processing device 110. In an embodiment, the inputcontroller 150 and the display device 140 are integrated into ahead-mounted display device and the input controller 150 comprises amotion sensor that detects position and orientation of the head-mounteddisplay device. The virtual perspective can then be derived tocorrespond to the position and orientation of the head-mounted displaydevice such that the virtual perspective corresponds to a perspectivethat would be seen by a viewer wearing the head-mounted display device.Thus, in this embodiment, the head-mounted display device can provide areal-time rendering of the scene as it would be seen by an observerwithout the head-mounted display. Alternatively, the input controller150 may comprise a user-controlled control device (e.g., a mouse,pointing device, handheld controller, gesture recognition controller,etc.) that enables a viewer to manually control the virtual perspectivedisplayed by the display device.

FIG. 2 illustrates an example embodiment of the mediated-reality system100 for a surgical application. Here, an embodiment of the camera array120 is positioned over the scene 130 (in this case, a surgical site) andcan be positioned via a swing arm 202 attached to a workstation 204. Theswing arm 202 may be manually moved or may be robotically controlled inresponse to the input controller 150. The display device 140 in thisexample is embodied as a virtual reality headset. The workstation 204may include a computer to control various functions of the camera array120 and the display device 140, and may furthermore include a secondarydisplay that can display a user interface for performing variousconfiguration functions, or may mirror the display on the display device140. The image processing device 120 and the input controller 150 mayeach be integrated in the workstation 204, the display device 140, or acombination thereof.

FIG. 3 illustrates a bottom plan view of an example embodiment of acamera array 120. The camera array 120 include a plurality of cells 202(e.g., four cells) each comprising one or more cameras 122. In anembodiment, the cells 202 each have a hexagonal cross-section and arepositioned in a honeycomb pattern. Particularly, two inner cells 202-A,202-B are each positioned adjacent to other cells 202 along threeadjacent edges, while two outer cells 202-C, 202-D are each positionedadjacent to other cells 202 along only two adjacent edges. The innercells 202-A, 202-B are positioned to have respective edges adjacent toeach other and may share a side wall, while the outer cells 202-C, 202-Dare separated from each other (are not in direct contact). Here, theouter cells 202-C, 202-D may each have a respective pair of edges thatare adjacent to respective edges of the inner cells 202-A, 202-B.Another feature of the illustrated cell arrangement is that the outercells 202-C, 202-D each include four edges that form part of the outerperimeter of the camera array 120 and the inner cells 202-A, 202-B eachinclude three edges that form part of the outer perimeter of the cameraarray 120.

The hexagonal shape of the cells 202 provides several benefits. First,the hexagonal shape enables the array 120 to be expanded to includeadditional cells 202 in a modular fashion. For example, while theexample camera array 120 includes four cells 202, other embodiments ofthe camera array 120 could include, for example eight or more cells 202by positioning additional cells 202 adjacent to the outer edges of thecells 202 in a honeycomb pattern. By utilizing a repeatable pattern,camera arrays 120 of arbitrary size and number of cameras 120 can bemanufactured using the same cells 202. Furthermore, the repeatablepattern can ensure that spacing of the cameras 122 is predictable, whichenables the image processor 120 to process images from different sizesof camera arrays 120 with different numbers of cameras 122 withoutsignificant modification to the image processing algorithms.

In an embodiment, the walls of the cells 202 are constructed of a rigidmaterial such as metal or a hard plastic. The cell structure providesstrong structural support for holding the cameras 122 in theirrespective positions without significant movement due to flexing orvibrations of the array structure.

In an embodiment, each cell 202 comprises a set of three cameras 122arranged in a triangle pattern with all cameras 122 oriented to focus ona single point. In an embodiment, each camera 122 is approximatelyequidistant from each of its neighboring cameras 122 within the cell 202and approximately equidistant from neighboring cameras 122 in adjacentcells 202. This camera spacing results in a triangular grid, where eachset of three neighboring cameras 122 are arranged in triangle ofapproximately equal dimensions. This spacing simplifies the processingperformed by the image processing device 110 when synthesizing theoutput image corresponding to the virtual camera perspective. Thetriangular grid furthermore allows for a dense packing of cameras 122within a limited area. Furthermore, the triangular grid enables thetarget volume to be captured with a uniform sampling rate to give smoothtransitions between camera pixel weights and low variance in generatedimage quality based on the location of the virtual perspective.

In an embodiment, each cell 202 comprises cameras 122 of at least twodifferent types. For example, in an embodiment, each cell 202 includestwo cameras 122-A of a first type (e.g., type A) and one camera 122-B ofa second type (e.g., type B). In an embodiment, the type A cameras 122-Aand the type B cameras 122-B have different focal lengths. For example,the type B cameras 122-B may have a shorter focal length than the type Acameras 122-A. In a particular example, the type A cameras 122-A have 50mm lenses while the type B cameras 122-B have 35 mm lenses. In anembodiment, the type B cameras 122-B are generally positioned in theirrespective cells 202 in the camera position furthest from a center pointof the array 120.

The type B cameras 122-B have a larger field-of-view and provide moreoverlap of the scene 130 than the type A cameras 122-A. The imagescaptured from these cameras 122-B are useful to enable geometryreconstruction and enlargement of the viewable volume. The type Acameras 122-A conversely have a smaller field-of-view and provide moreangular resolution to enable capture of smaller details than the type Bcameras 122-B. In an embodiment, the type A cameras occupy positions inthe center of the camera array 120 so that when points of interest inthe scene 150 (e.g., a surgical target) are placed directly below thecamera array 120, the captured images will benefit from the increaseddetail captured by the type A cameras 122-A relative to the type Bcameras 122-B. Furthermore, by positioning the type B cameras 122-Balong the exterior of the array 120, a wide baseline between the type Bcameras 122-B is achieved, which provides the benefit of enablingaccurate stereoscopic geometry reconstruction. For example, in the cells202-A, 202-C, 202-D, the type B camera 122-B is at the camera positionfurthest from the center of the array 120. In the case of a cell 202-Bhaving two cameras equidistant from the center point, one of the camerapositions may be arbitrarily selected for the type B camera 122-B. In analternative embodiment, the type B cameras 122-B may occupy the othercamera position equidistant from the center of the array 120.

In an embodiment, the camera array 120 further includes a projector 124that can project structured light onto the scene 130. The projector 124may be positioned near a center line of the camera array 120 in order toprovide desired coverage of the scene 130. The projector 124 may provideillumination and project textures and other patterns (e.g., to simulatea laser pointer or apply false or enhanced coloring to certain regionsof the scene 150). In an embodiment, the camera array 120 may alsoinclude depth sensors 126 adjacent to the projector 124 to use for depthestimation and object tracking.

FIG. 4 illustrates a more detailed bottom plan view of an embodiment ofa camera array 120. In this view, the orientation of the cameras can beseen as pointing towards a centrally located focal point. Furthermore,in this embodiment, the type A cameras 122-A are 50 mm focal lengthcameras and the type B cameras 122-B are 35 mm focal length cameras. Asfurther illustrated in this view, an embodiment of the camera array 120may include one or more cooling fans to provide cooling to the cameraarray 120. For example, in one embodiment, a pair of fans may bepositioned in the outer cells 202-C, 202-D of the camera array 120. Inan alternative embodiment, the camera array 120 may incorporateoff-board cooling via tubing that carries cool air to the camera array120 and/or warm air away from the camera array 120. This embodiment maybe desirable to comply with restrictions on airflow around a patient inan operating room setting.

FIG. 5 illustrates a perspective view of the camera array 120. In thisview, a top cover 504 is illustrated to cover the hexagonal cells 202and provide structural support to the camera array 120. Additionally,the top cover may include a mounting plate 506 for coupling to a swingarm 202 as illustrated in FIG. 2. The top cover 504 may further includemounting surfaces on the outer cells 202-C, 202-D for mounting the fans402.

Upon reading this disclosure, those of skill in the art will appreciatestill additional alternative structural and functional designs for thedisclosed embodiments as disclosed from the principles herein. Thus,while particular embodiments and applications have been illustrated anddescribed, it is to be understood that the disclosed embodiments are notlimited to the precise construction and components disclosed herein.Various modifications, changes and variations, which will be apparent tothose skilled in the art, may be made in the arrangement, operation anddetails of the method and system disclosed herein without departing fromthe scope of the described embodiments.

1. A camera array comprising: a plurality of hexagonal cells arranged ina honeycomb pattern in which a pair of inner cells include respectiveedges adjacent to each other and a pair of outer cells are separatedfrom each other by the inner cells; and a plurality of cameras mountedwithin each of the plurality of hexagonal cells, the plurality ofcameras including at least one camera of a first type and at least onecamera of a second type different than the first type, wherein theplurality of cameras within each of the plurality of hexagonal cells arearranged in a triangular grid approximately equidistant from neighboringcameras.
 2. The camera array of claim 1, wherein the at least one cameraof the second type within each of the plurality of hexagonal cells is ata position further from or equidistant from a center point of the cameraarray relative to cameras of the first type.
 3. The camera array ofclaim 1, wherein the at least one camera of the first type has a longerfocal length than the at least one camera of the second type.
 4. Thecamera array of claim 1, wherein the outer cells each include four edgesalong an exterior perimeter of the camera array and wherein the innercells each include three edges along the exterior perimeter of thecamera array.
 5. The camera array of claim 1, wherein the camera of thefirst type has a narrower field of view than the camera of the secondtype.
 6. The camera array of claim 1, wherein the camera of the firsttype comprises a 50 mm camera and wherein the camera of the second typecomprises a 35 mm camera.
 7. The camera array of claim 1, furthercomprising: a projector to project structured light onto a scene thatwithin a field of view of the plurality of cameras.
 8. The camera arrayof claim 1, further comprising: a depth sensor for sensing depth of asurface of a scene within a field of view of the plurality of cameras.9. The camera array of claim 1, further comprising: a cooling system toprovide cooling to the plurality of cameras.
 10. The camera array ofclaim 1, further comprising a swing arm to position the plurality ofcameras in a desired position and orientation.
 11. A mediated-realitysystem, comprising: a camera array to capture a plurality of images of ascene, the camera array including: a plurality of hexagonal cellsarranged in a honeycomb pattern in which a pair of inner cells includerespective edges adjacent to each other and a pair of outer cells areseparated from each other by the inner cells; and a plurality of camerasmounted within each of the plurality of hexagonal cells, the pluralityof cameras including at least one camera of a first type and at leastone camera of a second type different than the first type, wherein theplurality of cameras within each of the plurality of hexagonal cells arearranged in a triangular grid approximately equidistant from neighboringcameras; an input controller to control a position and orientation of avirtual perspective of the scene; and an image processing device tosynthesize a virtual image corresponding to the virtual perspectivebased on the plurality of images of the scene; and a display device todisplay the virtual image.
 12. The mediated-reality system of claim 11,wherein the at least one camera of the second type within each of theplurality of hexagonal cells is at a position further from orequidistant from a center point of the camera array relative to camerasof the first type.
 13. The mediated-reality system of claim 11, whereinthe at least one camera of the first type has a longer focal length thanthe at least one camera of the second type.
 14. The mediated-realitysystem of claim 11, wherein the outer cells each include four edgesalong an exterior perimeter of the camera array and wherein the innercells each include three edges along the exterior perimeter of thecamera array.
 15. The mediated-reality system of claim 11, wherein thecamera of the first type has a narrower field of view than the camera ofthe second type.
 16. The mediated-reality system of claim 11, whereinthe camera of the first type comprises a 50 mm camera and wherein thecamera of the second type comprises a 35 mm camera.
 17. Themediated-reality system of claim 11, further comprising: a projector toproject structured light onto a scene that within a field of view of theplurality of cameras.
 18. A camera array comprising: a plurality ofcells; and a plurality of cameras mounted within each of the pluralityof cells, the plurality of cameras including at least one camera of afirst type and at least one camera of a second type different than thefirst type, wherein the plurality of cameras within each of theplurality of cells are arranged in a triangular grid approximatelyequidistant from neighboring cameras.
 19. The camera array of claim 18,wherein the at least one camera of the second type within each of theplurality of hexagonal cells is at a position further from orequidistant from a center point of the camera array relative to camerasof the first type.
 20. The camera array of claim 18, wherein the atleast one camera of the first type has a longer focal length than the atleast one camera of the second type.